{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Binary Image Denoising"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "*Jiaolong Xu (GitHub ID: [Jiaolong](https://github.com/Jiaolong))*"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "This notebook is written during GSoC 2014. Thanks Shell Hu and Thoralf Klein for taking time to help me on this project!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook illustrates how to use shogun structured output learning framework for binary images denoising. The task is defined as a pairwise factor graph model with [Graph cuts](http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CGraphCut.html) inference, where model parameters are learned by SOSVM using a [SGD solver](http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CStochasticSOSVM.html)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook illustrates how to use shogun structured output learning framework for binary images denoising. I recommend [1] for a nice introduction of structured learning and prediction in computer vision. One of the founding publications on the topic of learning structured models might be [4]. In the following, I will give an explicit example of structured output prediction for binary image denoising.\n",
      "\n",
      "Given a noise black/withe image $\\textbf{x}$ of size $m \\times n$, the task of denoising is to predicted the original binary image $\\textbf{y}$. We flatten the image into a long vector $\\textbf{y} = [y_0, \\dots, y_{m\\times n}]$, where $y_i \\in \\{0, 1\\}$ is the value for pixel $i$. In this work, we aim to learn a model from a bunch of noise input images and their ground truth binary images, i.e., supervised learning. One may think about learning a binary classifier for each pixel. It has several drawbacks. First, we may not care about classifying every single pixel completely correct, i.e., if we misclassify a single pixel, this is not as bad as misclassify a whole image. Second, we lose all context, e.g., pairwise pixels (one pixel and its neighbore). The structured predition here is to predict an entire binary image $\\textbf{y}$ or a grid graph of $m \\times n$. Here, the output space $\\mathcal{Y}$ is all possible binary images of size $m \\times n$. It can be formulated as following:\n",
      "\n",
      "$$\n",
      "\\hat{\\textbf{y}} = \\underset{\\textbf{y} \\in \\mathcal{Y}}{\\operatorname{argmax}} f(\\textbf{x},\\textbf{y}),    (1)\n",
      "$$\n",
      "\n",
      "where $f(\\textbf{x},\\textbf{y})$ is the compitibility function, measures how well $\\textbf{y}$ fits $\\textbf{x}$. There are basically three challenges in doing structured learning and prediction:\n",
      "- Choosing a parametric form of $f(\\textbf{x},\\textbf{y})$\n",
      "- solving $\\underset{\\textbf{y} \\in \\mathcal{Y}}{\\operatorname{argmax}} f(\\textbf{x},\\textbf{y})$\n",
      "- learning parameters for $f(\\textbf{x},\\textbf{y})$ to minimize a loss\n",
      "\n",
      "In this work, our parameters are pairwise and unary potentials and they can be written as:\n",
      "\n",
      "$$\n",
      "f(\\textbf{x},\\textbf{y}) = \\sum_i \\textbf{w}_i'\\phi_i(\\textbf{x}) + \\sum_{i,j} \\textbf{w}_{ij}'\\phi_{ij}(\\textbf{x}),    (2)\n",
      "$$\n",
      "\n",
      "where $\\textbf{w}_i$ and $\\textbf{w}_{ij}$ are unary and pairwise parameters, $\\phi_i(\\textbf{x})$ and $\\phi_{ij}(\\textbf{x})$ are unary and pairwise features respectively. Equation (2) is a linear function and can be written as a dot product of a global parameter $\\textbf{w}$ and joint feature vector $\\Phi(\\textbf{x},\\textbf{y})$, i.e.,  $f(\\textbf{x},\\textbf{y}) = \\textbf{w}'\\Phi(\\textbf{x}, \\textbf{y})$. The global parameter $\\textbf{w}$ is a collection of unary and pairwise parameters. The joint feature $\\Phi(\\textbf{x}, \\textbf{y})$ maps local features, e.g., pixel values from each location, to the corresponding location of the global feature vector according to $\\textbf{y}$. In factor graph model, parameters are associated with a set of factor types. \n",
      "\n",
      "As said before, the output space $\\mathcal{Y}$ is usually finite but very large. In our case, it is all possible binary images of size $m \\times n$. Finding ${\\operatorname{argmax}}$ in such a large space by exhaustive search is not practical. To do the maximization over $\\textbf{y}$ efficiently, the most popular tool is using energy functions or conditional random fields (CRFs). In this work, we implemented Graph cuts [5] for efficient inference. We also implemented max-product LP relaxation inference and tree max-product inference. However, the later is limited to tree-struct graph while for image denosing, we use grid graph.\n",
      "\n",
      "The parameters are learned by regularized risk minimization, where the risk defined by user provided loss function $\\Delta(\\mathbf{y},\\mathbf{\\hat{y}})$. We use the Hamming loss in this experiment. The empirical risk is defined in terms of the surrogate hinge loss $\\mathcal{L}_i(\\mathbf{w}) = \\max_{\\mathbf{y} \\in \\mathcal{Y}} \\Delta(\\mathbf{y}_i,\\mathbf{y}) - \\mathbf{w}' [\\Phi(\\mathbf{x}_i,\\mathbf{y}_i) - \\Phi(\\mathbf{x}_i,\\mathbf{y})]$. The training objective is given by\n",
      "\n",
      "$$\n",
      "\\min_{\\mathbf{w}} \\frac{\\lambda}{2} ||\\mathbf{w}||^2 + \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\mathbf{w}).     (3)\n",
      "$$"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Create binary denoising dataset"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "import numpy.random"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We dfine an Example class for the training and testing examples."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class Example:\n",
      "    \"\"\" Example class.\n",
      "    \n",
      "        Member variables:\n",
      "        id: id of the example\n",
      "        im_bw: original binary image\n",
      "        im_noise: original image with noise\n",
      "        feats: feature for each pixel\n",
      "        labels: binary labels of each pixel\n",
      "    \"\"\"\n",
      "    \n",
      "    def __init__(self, id, im_bw, im_noise, feats, labels):\n",
      "        self.id = id\n",
      "        self.im_bw = im_bw\n",
      "        self.im_noise = im_noise\n",
      "        self.feature = feats\n",
      "        self.labels = labels"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In the following, we create a toy dataset. Similar to [2], we make random noisy images, then smooth them to make the true (discrete) output values. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import scipy.ndimage\n",
      "# generate random binary images\n",
      "im_size = np.array([50, 50], np.int32)\n",
      "num_images = 30\n",
      "\n",
      "ims_bw = []\n",
      "for i in range(num_images):\n",
      "        im_rand = np.random.random_sample(im_size)\n",
      "        im_bw = np.round(scipy.ndimage.gaussian_filter(im_rand, sigma=3))\n",
      "        ims_bw.append(im_bw)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, noises are added to the binary images. We apply the same strategy as in [3], the noisy images are generated as $z_i = x_i(1-t_i^n) + (1-x_i)t_i^n$, where $x_i$ is the true binary label, and $t_i \\in [0,1]$ is a random value. Here, $n \\in (1, \\infty)$ is the noise level, where lower values correspond to more noise.\n",
      "\n",
      "In this experiment, we use only two features as unary features: a constant of $1$ and the noisy input value at the pixel, i.e., $\\textbf{u}(i) = [z_i, 1]$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# define noisy level\n",
      "noise_level = 2\n",
      "\n",
      "# initialize an empty list\n",
      "example_list = []\n",
      "\n",
      "for i in range(len(ims_bw)):\n",
      "    im_bw = ims_bw[i]\n",
      "    # add noise to the binary image\n",
      "    t = np.random.random_sample(im_bw.shape)\n",
      "    im_noise = im_bw*(1-t**noise_level) + (1-im_bw)*(t**noise_level)\n",
      "    \n",
      "    # create 2-d unary features\n",
      "    c1 = np.ravel(im_noise)\n",
      "    c2 = np.ones(im_noise.size, np.int32)\n",
      "    feats = np.column_stack([c1, c2])\n",
      "    # we use pixel-level labels\n",
      "    # so just flatten the original binary image into a vector\n",
      "    labels = np.ravel(im_bw)\n",
      "    example = Example(i, im_bw, im_noise, feats, labels)\n",
      "    example_list.append(example)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we creat a function to visualize our examples."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def plot_example(example):\n",
      "    \"\"\" Plot example.\"\"\"\n",
      "    \n",
      "    fig, plots = plt.subplots(1, 2, figsize=(12, 4))\n",
      "    plots[0].matshow(example.im_bw, cmap=plt.get_cmap('Greys'))\n",
      "    plots[0].set_title('Binary image')\n",
      "    plots[1].matshow(example.im_noise, cmap=plt.get_cmap('Greys'))\n",
      "    plots[1].set_title('Noise image')\n",
      "    \n",
      "    for p in plots:\n",
      "        p.set_xticks(())\n",
      "        p.set_yticks(())\n",
      "\n",
      "    plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# plot an example\n",
      "plot_example(example_list[9])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Build Factor Graph Model"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from modshogun import Factor, TableFactorType, FactorGraph\n",
      "from modshogun import FactorGraphObservation, FactorGraphLabels, FactorGraphFeatures\n",
      "from modshogun import FactorGraphModel, GRAPH_CUT, LP_RELAXATION\n",
      "from modshogun import MAPInference"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We define a 'make_grid_edges' function to compute the indeces of the pairwise pixels. we use grid graph with neighborhood size of $4$ in our experiment."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def make_grid_edges(grid_w, grid_h, neighborhood=4):\n",
      "    \"\"\" Create grid edge lists.\n",
      "    \n",
      "        Args:\n",
      "            grid_w: width of the grid\n",
      "            grid_h: height of the grid\n",
      "            neigborhood: neigborhood of the node (4 or 8)\n",
      "        \n",
      "        Returns:\n",
      "            edge list of the grid graph\n",
      "    \"\"\"\n",
      "    if neighborhood not in [4, 8]:\n",
      "        raise ValueError(\"neighborhood can only be '4' or '8', got %s\" % repr(neighborhood))\n",
      "    inds = np.arange(grid_w * grid_h).reshape([grid_w, grid_h])\n",
      "    inds = inds.astype(np.int64)\n",
      "    right = np.c_[inds[:, :-1].ravel(), inds[:, 1:].ravel()]\n",
      "    down = np.c_[inds[:-1, :].ravel(), inds[1:, :].ravel()]\n",
      "    edges = [right, down]\n",
      "    if neighborhood == 8:\n",
      "        upright = np.c_[inds[1:, :-1].ravel(), inds[:-1, 1:].ravel()]\n",
      "        downright = np.c_[inds[:-1, :-1].ravel(), inds[1:, 1:].ravel()]\n",
      "        edges.extend([upright, downright])\n",
      "        \n",
      "    return np.vstack(edges)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# in this experiment, we use fixed image size\n",
      "im_w = example_list[0].im_bw.shape[1]\n",
      "im_h = example_list[0].im_bw.shape[0]\n",
      "# we compute the indeces of the pairwise nodes\n",
      "edge_list = make_grid_edges(im_w, im_h)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For binary denosing, we define two types of factors:\n",
      "- unary factor: the unary factor type is used to define unary potentials that captures the the appearance likelyhood of each pixel. We use very simple unary feature in this experiment, the pixel value and a constant value $1$. As we use binary label, thus the size of the unary parameter is $4$.\n",
      "- pairwise factor: the pairwise factor type is used to define pairwise potentials between each pair of pixels. There features of the pairwise factors are constant $1$ and there are no additional edge features. For the pairwise factors, there are  $2 \\times 2$ parameters.\n",
      "\n",
      "Putting all parameters together, the global parameter vector $\\mathbf{w}$ has length $8$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def define_factor_type(num_status, dim_feat):\n",
      "    \"\"\" Define factor type.\n",
      "\n",
      "        Args:\n",
      "            num_status: number of status.\n",
      "            dim_feat: dimention of the unary node feature\n",
      "\n",
      "        Returns:\n",
      "            ftype_unary: unary factor type\n",
      "            ftype_pair: pairwise factor type\n",
      "    \"\"\"\n",
      "\n",
      "    # unary, type id = 0\n",
      "    cards_u = np.array([num_status], np.int32) # cardinalities\n",
      "    w_u = np.zeros(num_status*dim_feat, np.float64)\n",
      "    ftype_unary = TableFactorType(0, cards_u, w_u)\n",
      "\n",
      "    # pairwise, type id = 1\n",
      "    cards_p = np.array([num_status, num_status], np.int32)\n",
      "    w_p = np.zeros(num_status*num_status, np.float64)\n",
      "    ftype_pair = TableFactorType(1, cards_p, w_p)\n",
      "\n",
      "    return ftype_unary, ftype_pair"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# define factor types\n",
      "ftype_unary, ftype_pair = define_factor_type(num_status=2, dim_feat=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def prepare_factor_graph_model(example_list, ftype_unary, ftype_pair, edge_list, num_status = 2, dim_feat = 2):\n",
      "    \"\"\" Prepare factor graph model data.\n",
      "\n",
      "        Args:\n",
      "            example_list: the examples\n",
      "            num_status: number of status\n",
      "            dim_feat: dimention of the unary features\n",
      "    \"\"\"\n",
      "\n",
      "    num_samples = len(example_list)\n",
      "    # Initialize factor graph features and labels\n",
      "    feats_fg = FactorGraphFeatures(num_samples)\n",
      "    labels_fg = FactorGraphLabels(num_samples)\n",
      "\n",
      "    # Interate over all the examples\n",
      "    for i in range(num_samples):\n",
      "        example = example_list[i]\n",
      "        feats = example.feature\n",
      "        num_var = feats.shape[0]\n",
      "        dim_feat = feats.shape[1]\n",
      "        # Initialize factor graph\n",
      "        cards = np.array([num_status]*num_var, np.int32) # cardinalities\n",
      "        fg = FactorGraph(cards)\n",
      "\n",
      "        # add unary\n",
      "        for u in range(num_var):\n",
      "            data_u = np.array(feats[u,:], np.float64)\n",
      "            inds_u = np.array([u], np.int32)\n",
      "            factor_u = Factor(ftype_unary, inds_u, data_u)\n",
      "            fg.add_factor(factor_u)\n",
      "        \n",
      "        # add pairwise\n",
      "        for p in range(edge_list.shape[0]):\n",
      "            data_p = np.array([1.0], np.float64)\n",
      "            inds_p = np.array(edge_list[p,:], np.int32)\n",
      "            factor_p = Factor(ftype_pair, inds_p, data_p)\n",
      "            fg.add_factor(factor_p)\n",
      "        \n",
      "        # add factor graph feature\n",
      "        feats_fg.add_sample(fg)\n",
      "        # add factor graph label\n",
      "        labels = example.labels.astype(np.int32)\n",
      "        assert(labels.shape[0] == num_var)\n",
      "        loss_weight = np.array([1.0/num_var]*num_var)\n",
      "        f_obs = FactorGraphObservation(labels, loss_weight)\n",
      "        labels_fg.add_label(f_obs)\n",
      "\n",
      "    return feats_fg, labels_fg"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We split the samples into training and testing sets. The features and labels are converted for factor graph model."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "num_train_samples = 10\n",
      "\n",
      "examples_train = example_list[:num_train_samples]\n",
      "examples_test = example_list[num_train_samples:]\n",
      "\n",
      "# create features and labels for factor graph mode\n",
      "(feats_train, labels_train) = prepare_factor_graph_model(examples_train, ftype_unary, ftype_pair, edge_list)\n",
      "(feats_test, labels_test) = prepare_factor_graph_model(examples_test, ftype_unary, ftype_pair, edge_list)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this experiment, we use [Graph cuts](http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CGraphCut.html) as approximate inference algorithm, i.e., solve Eq. (1). Please refer to [4] for a comprehensive understanding."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# inference algorithm\n",
      "infer_alg = GRAPH_CUT\n",
      "#infer_alg = LP_RELAXATION"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create model and register factor types\n",
      "model = FactorGraphModel(feats_train, labels_train, infer_alg)\n",
      "model.add_factor_type(ftype_unary)\n",
      "model.add_factor_type(ftype_pair)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Learning parameter with structured output SVM"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We apply (<a href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CStochasticSOSVM.html\">StochasticSOSVM</a>) to learn the parameter $\\textbf{w}$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from modshogun import StochasticSOSVM\n",
      "import time\n",
      "\n",
      "# Training with Stocastic Gradient Descent\n",
      "sgd = StochasticSOSVM(model, labels_train, True, True)\n",
      "sgd.set_num_iter(300)\n",
      "sgd.set_lambda(0.0001)\n",
      "\n",
      "# train\n",
      "t0 = time.time()\n",
      "sgd.train()\n",
      "t1 = time.time()\n",
      "w_sgd = sgd.get_w()\n",
      "print \"SGD took\", t1 - t0, \"seconds.\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def evaluation(labels_pr, labels_gt, model):\n",
      "    \"\"\" Evaluation\n",
      "\n",
      "        Args:\n",
      "            labels_pr: predicted label\n",
      "            labels_gt: ground truth label\n",
      "            model: factor graph model\n",
      "\n",
      "        Returns:\n",
      "            ave_loss: average loss\n",
      "    \"\"\"\n",
      "\n",
      "    num_train_samples = labels_pr.get_num_labels()\n",
      "    acc_loss = 0.0\n",
      "    ave_loss = 0.0\n",
      "    for i in range(num_train_samples):\n",
      "        y_pred = labels_pr.get_label(i)\n",
      "        y_truth = labels_gt.get_label(i)\n",
      "        acc_loss = acc_loss + model.delta_loss(y_truth, y_pred)\n",
      "\n",
      "    ave_loss = acc_loss / num_train_samples\n",
      "\n",
      "    return ave_loss"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# training error\n",
      "labels_train_pr = sgd.apply()\n",
      "ave_loss = evaluation(labels_train_pr, labels_train, model)\n",
      "print('SGD: Average training error is %.4f' % ave_loss)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def plot_primal_trainError(sosvm, name = 'SGD'):\n",
      "    \"\"\" Plot primal objective values and training errors.\"\"\"\n",
      "    \n",
      "    primal_val = sosvm.get_helper().get_primal_values()\n",
      "    train_err = sosvm.get_helper().get_train_errors()\n",
      "    \n",
      "    fig, plots = plt.subplots(1, 2, figsize=(12,4))\n",
      "    \n",
      "    # primal vs passes\n",
      "    plots[0].plot(range(primal_val.size), primal_val, label=name)\n",
      "    plots[0].set_xlabel('effecitve passes')\n",
      "    plots[0].set_ylabel('primal objective')\n",
      "    plots[0].set_title('whole training progress')\n",
      "    plots[0].legend(loc=1)\n",
      "    plots[0].grid(True)\n",
      "    \n",
      "    # training error vs passes\n",
      "    plots[1].plot(range(train_err.size), train_err, label=name)\n",
      "    plots[1].set_xlabel('effecitve passes')\n",
      "    plots[1].set_ylabel('training error')\n",
      "    plots[1].set_title('effective passes')\n",
      "    plots[1].legend(loc=1)\n",
      "    plots[1].grid(True) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# plot primal objective values and training errors at each pass\n",
      "plot_primal_trainError(sgd)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Testing results"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Testing error\n",
      "sgd.set_features(feats_test)\n",
      "sgd.set_labels(labels_test)\n",
      "\n",
      "labels_test_pr = sgd.apply()\n",
      "ave_loss = evaluation(labels_test_pr, labels_test, model)\n",
      "print('SGD: Average testing error is %.4f' % ave_loss)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def plot_results(example, y_pred):\n",
      "    \"\"\" Plot example.\"\"\"\n",
      "    \n",
      "    im_pred = y_pred.reshape(example.im_bw.shape)\n",
      "    \n",
      "    fig, plots = plt.subplots(1, 3, figsize=(12, 4))\n",
      "    \n",
      "    plots[0].matshow(example.im_noise, cmap=plt.get_cmap('Greys'))\n",
      "    plots[0].set_title('noise input')\n",
      "    plots[1].matshow(example.im_bw, cmap=plt.get_cmap('Greys'))\n",
      "    plots[1].set_title('ground truth labels')\n",
      "    plots[2].matshow(im_pred, cmap=plt.get_cmap('Greys'))\n",
      "    plots[2].set_title('predicted labels')\n",
      "    \n",
      "    for p in plots:\n",
      "        p.set_xticks(())\n",
      "        p.set_yticks(())\n",
      "\n",
      "    plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline\n",
      "\n",
      "# plot one example\n",
      "i = 8\n",
      "\n",
      "# get predicted output\n",
      "y_pred = FactorGraphObservation.obtain_from_generic(labels_test_pr.get_label(i)).get_data()\n",
      "\n",
      "# plot results\n",
      "plot_results(examples_test[i], y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def plot_results_more(examples, labels_pred, num_samples=10):\n",
      "    \"\"\" Plot example.\"\"\"\n",
      "    \n",
      "    fig, plots = plt.subplots(num_samples, 3, figsize=(12, 4*num_samples))\n",
      "    \n",
      "    for i in range(num_samples):\n",
      "        example = examples[i]\n",
      "        \n",
      "        # get predicted output\n",
      "        y_pred = FactorGraphObservation.obtain_from_generic(labels_pred.get_label(i)).get_data()\n",
      "        im_pred = y_pred.reshape(example.im_bw.shape)\n",
      "    \n",
      "        plots[i][0].matshow(example.im_noise, cmap=plt.get_cmap('Greys'))\n",
      "        plots[i][0].set_title('noise input')\n",
      "        plots[i][0].set_xticks(())\n",
      "        plots[i][0].set_yticks(())\n",
      "        \n",
      "        plots[i][1].matshow(example.im_bw, cmap=plt.get_cmap('Greys'))\n",
      "        plots[i][1].set_title('ground truth labels')\n",
      "        plots[i][1].set_xticks(())\n",
      "        plots[i][1].set_yticks(())\n",
      "        \n",
      "        plots[i][2].matshow(im_pred, cmap=plt.get_cmap('Greys'))\n",
      "        plots[i][2].set_title('predicted labels')\n",
      "        plots[i][2].set_xticks(())\n",
      "        plots[i][2].set_yticks(())\n",
      "\n",
      "    plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plot_results_more(examples_test, labels_test_pr, num_samples=5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Reference"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[1] Nowozin, S., & Lampert, C. H. Structured learning and prediction in computer vision. Foundations and Trends\u00ae in Computer Graphics and Vision, 6(3\u20134), 185-365, 2011."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[2] http://users.cecs.anu.edu.au/~jdomke/JGMT/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[3] Justin Domke, Learning Graphical Model Parameters with Approximate Marginal Inference, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 10, pp. 2454-2467, 2013. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[4] Tsochantaridis, I., Hofmann, T., Joachims, T., Altun, Y., Support Vector Machine Learning for Interdependent and Structured Ouput Spaces, ICML 2004."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[5] Boykov, Y., Veksler, O., & Zabih, R. Fast approximate energy minimization via graph cuts. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 23(11), 1222-1239, 2001."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}